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ABSTRACT 



This paper describes a program to measure institutional 



effectiveness that involved development of an inventory checklist designed to 
stimulate faculty input on how student performance- related outcomes are 
measured and to evaluate current program effectiveness. The checklist, 
designed to be administered in an interview format, focused on three 
particular areas: (1) departmental mission, intended educational outcomes, 

and written methods of assessment for evaluating program effectiveness; (2) 
direct indicators for assessing students' knowledge and skills, including 
capstone courses, portfolio assessments, licensure, certification, 
professional exams; and indirect indicators for assessing students' and 
others' opinions of their learning, including student course evaluations, 
employer surveys, student exit interviews, and alumni surveys; and (3) using 
inventory results to identify needed program improvements and additional 
resources such as training, personnel, and technology. The paper concludes 
that using the inventory checklist provides a non- threatening way to assess 
student outcomes, and offers an excellent opportunity for initiating a 
systematic and continuing process for institutional effectiveness and 
educational improvement. The checklist is appended. (Contains 12 references.) 
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Finding the ‘Start Line’ with an Institutional 



Effectiveness Inventory 



Although the assessment movement is approaching the quarter-century mark, there are 
still outposts where the use of student performance-related outcomes has not entered the 
mainstream of institutional life. In order to open the conversation on assessment, we 
designed a checklist to inventory current campus practices. The checklist, designed to be 
administered in an interview format, gathered information on intended educational 
outcomes, existing and potential assessment techniques and instruments. Results from 
the inventories were assembled into a diagnostic profile that identified resources, needs, 
and areas where centrally directed assessment activities would benefit all departments 
and programs. 



It has been nearly a quarter of a century since assessment began as a way for a 
college or university to measure its impact on students. Instead of focusing on inputs and 
resources, assessment looks critically at student learning, articulating program goals, 
setting standards for accomplishment, and measuring the extent to which standards are 
met. The question, “What should our graduates know, be able to do, and value?” is the 
focal point for beginning an assessment effort where outcomes matter most. 

The mandate for assessment developed as various national commissions studying 
higher education concluded that a “disturbing and dangerous mismatch exists between 
what American society needs from higher education and what it is receiving.” 
(Wingspread Group on Higher Education, 1993). What began as an experimental 
response to those concerns on the part of a handful of institutions has developed into a 
national agenda. For the past 15 years, there has been relentless pressure for assessment 
and the expectations associated with it (Gray, 1997). 

Influences on student assessment have been exerted by national-level efforts, 
state-level initiatives, accreditation agencies, the private sector and professional 
associations. All six regional accrediting agencies now require evidence of student 
assessment. According to a recent survey, all but four of fifty states reported some type 
of student assessment, and the requirement to demonstrate effectiveness via student 
outcomes has become part of performance funding in a number of states (Ewell, 1996; 
Cole, Nettles &, Sharp, 1997; Peterson and Einarson, 2000). Regardless of who guides 
the effort to strengthen institutional accoimtability, it appears inevitable that student 
assisted with data analysis for this paper. 
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outcomes assessment will endure as a topic of institutional, regional and national concern 
into the 21*‘ century. 

Conceptual Framework 

Like many institutions, our impetus for instituting a program of institutional 
effectiveness was an impending regional accreditation visit. Undertaking assessment in 
at atmosphere where many faculty are unfamiliar with the concept is a challenge, 
particularly at a large, diffuse umversity where colleges are independent and faculty in a 
single discipline may be scattered across several campuses. In order to lay the 
groundwork for an initiative that is certain to claim a fair amount of time and resources 
over the next several years, we decided to begin by polling departments on their existing 
assessment activities. We chose to develop an inventory checklist to identify current 
campus practices for assessing student performance-related outcomes. 

The development of an inventory checklist was not a novel idea for such an 
undertaking. The Southern Association of Colleges and Schools encourages its member 
institutions to begin their evaluation of institutional effectiveness by conducting an 
assessment of existing practices within the institution, and even suggests a format for the 
inventory (Commission on Colleges of the Southern Association of Colleges and 
Schools, 1996). Nichols (1995) discusses the importance and added value of conducting 
an inventory of assessment activities. Many of his case study institutions reported the use 
of an inventory for means of collecting and reporting information (Nichols, 1993). 
Palomba and Banta (1999) point out that completing an inventory of current data 
collection methods may uncover several activities that, although not specifically designed 
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for program assessment, can help accomplish its purpose. Institutions using inventories 
reported benefits ranging from the identification of disparities in existing operations and 
programs to using inventory results as benchmarks to measure future assessment efforts 
(Nichols, 1995; Hodge, 1997). 

Administering the Inventory Checklist 

The purpose of the inventory checklist was to stimulate conversation and provoke 
thought among department chairs and their units about ways in which student 
performance-related outcomes are measured and to evaluate current methods for overall 
program effectiveness. The inventory checklist identified strengths, weaknesses, and 
needed resources in individual programs. It highlighted areas in which programs had 
proven methods of assessment in place, and uncovered methods of assessment not in 
place but potentially useful. It allowed respondents to determine which methods were not 
desirable or applicable for program assessment. 

The inventory checklist was designed to be administered in an interview setting 
and with three areas of interest in mind. The first area focused on departmental mission, 
intended educational outcomes, and written methods of assessment for evaluating 
program effectiveness. The second area focused on direct and indirect indicators of 
assessment. Direct indicators assess knowledge and skills demonstrated by the student, 
such as capstone courses, portfolio assessments, licensure, certification, or professional 
exams, and video and audiotape evaluations. Indirect indicators assess students’ and 
others’ opinions of their learning, such as student course evaluations, employer surveys 
and questionnaires, student exit interviews, and alumni surveys. The third area focused 
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on the use of inventory results for overall program improvement, and identification of 
needed resources such as training, personnel, and technology, for improving student 
outcomes and program effectiveness. 

Prior to the interview, a web search was conducted to identify educational 
outcomes plans from other institutions that had completed, or were in the process of 
completing, the SACS reaffirmation Self-Study. At least one educational outcomes plan 
was printed for each discipline, so that chairs could have a tangible example of 
assessment work done by their colleagues in other institutions. The plan, along with a 
copy of the inventory checklist, was provided to each department chair a few days prior 
to the interview. 

The interview consisted of a one-on-one discussion with each department chair. 
The focus of the interview was to complete the inventory checklist, gather any supporting 
documentation offered, and ask as many questions as possible about specific outcomes 
assessment methods used by each department or program. The inventory checklist 
proved beneficial in gathering information in several ways. First, it stimulated an 
exchange of information between the department chair and the interviewer regarding 
specific outcomes assessment methods used by departments and programs. Second, it 
delineated examples of many different types of outcomes assessment methods, thus 
encouraging departments and programs to consider assessment in broader, and less 
strictly quantitative, terms. Third, it brought to realization the fact that some departments 
and programs were well on their way in identifying formal assessment activities. 
Likewise, departments or programs lagging in formal assessment activities were 



ERIC 



7 



4 



identified. Finally, departments and programs were able to identify and prioritize the 
areas where programs, services, and operations needed to be improved. 

The inventory checklist, coupled with individual interviews, proved to be an 
effective method for gathering information about individual departments and programs. 
The interview format was non-threatening, thereby allowing for an open and honest 
dialogue between the interviewer and the department chair. Department chairs candidly 
voiced their concerns, fears, and support about the reaffirmation project. Two-way 
communications between the interviewer and the department chair was not only 
stimulated, but also encouraged. Each party was given the opportunity to ask questions, 
not only concerning the department or program, but the reaffirmation project as well. 

Administering the checklist to the administrative and academic support units took 
a slightly different route. Believing that it was impractical and unnecessary to interview 
each unit director separately, we instead attended scheduled staff meetings in each area 
and explained the institutional effectiveness cycle, walking the participants through the 
checklist. We asked for the inventories to be completed and returned to our office within 
a month. The response rate was good, but the self-report aspect of this collection effort 
made the results a little less credible, particularly where documentation was not attached 
to the checklist. 

Using the Checklist Results 

Information front the completed checklists was first organized onto a spreadsheet 
to simplify analysis of results (Figure 1). Tallied results indicated that 80% of the 
departments reported having written mission statements, 43% said that they had 
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Figure 1 , Sample spreadsheet data for compiling checklist results. 



College 


Business 


Business 


Business 


Business Business 


Department 


Marketing 


School of Accounting 


Economics 


Finance 


Program 


Marketing 


Accounting 


Economics 


Finance 


Program level 


BS 


MACC 


MAT 


BS 


Written mission statement? 


Yes 


Yes 


Yes 


Yes 


Intended educational outcomes? 


Yes 


Yes 


No 


Yes 


Written methods of assessment? 


Yes 


Yes 


No 


Yes 


Separate accreditation? 


AACSB 


No 


AACSB 


AACSB 


Direct indicators of assessment: 










Comprehensive exam 


Currently use 


N/A 


N/A 


Would use 


Writing proficiency exam 


N/A 


Currently use 


N/A 


N/A 


National exam 


Would use 


N/A 


Would use 


N/A 


GRE subject test 


N/A 


N/A 


N/A 


N/A 


Certification exam 


N/A 


Currently use 


N/A 


N/A 


Licensure exam 


N/A 


Currently use 


N/A 


N/A 


Local pretest-posttest 


Would use 


N/A 


Would use 


N/A 


Performance assessment 


N/A 


N/A 


N/A 


N/A 


Video/audio tape evaluation 


N/A 


N/A 


N/A 


N/A 


Senior thesis/major project 


N/A 


Currently use 


N/A 


Currently use 


Portfolio evaluation 


N/A 


N/A 


N/A 


N/A 


Capstone courses 


Would use 


N/A 


N/A 


Currently use 



articulated intended educational outcomes, and 37% had defined methods of assessment 
for evaluating program effectiveness in terms of measurable student outcomes. Almost 
all reported that they had used assessment, however informally obtained, to improve 
programs. 

The ordering of direct measures shown in Figure 2 clarified which indicators were 
most frequently used across departments. Direct measures are those that require students 
to display their knowledge and skills as they respond to the measurement itself Senior 
projects topped the list of most frequently used direct indicators, followed by the GRE 
subject tests, capstone courses and comprehensive exams. The “white space” between 
shaded bars was of special interest, since it points to indicators departments would be 
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willing to consider making a part of their program assessments. For the most part, white 
space was scarce for direct indicators; departments were either using the indicator or 
were not interested in using it. Measures requiring performance rather than testing were 
slightly more popular. Since fewer than half of the departments were either using or 
planning to use most direct measures, it became obvious that these measures were going 
to be a hard sell. 

The assessment picture for indirect indicators looked more hopeful (Figure 3). 
Indirect measures ask students, or those connected with them, to reflect on their learning 
rather than demonstrate it. Aside from course evaluations, which most departments 
engage in regularly, there was low participation but high interest in using most of the 
indirect indicators. These generally fell into two categories; those involving analysis of 
existing data (“Analysis of grade distribution, “ “Examination of department data,” 
“Comparison with peer institutions”) and those requiring collection of outcomes data on 
graduates. Having an empirical identification of these departmental needs gave our 
office the mandate we felt was necessary to proceed with centrally directed assessment 
activities. 

Arraying the indicators by college provided a more detailed diagnosis of 
assessment activities and needs. For example. Figure 4 illustrates that most departments 
in the College of Arts and Letters had little interest in direct measures. Although 
licensure and certification exams are not used in these disciplines, the rejection of almost 
any kind of direct display of knowledge and skills was disconcerting. More favorable 
was the interest displayed in using indirect indicators. The patterns within as well as 
across the columns quickly profiled those departments likely to be cooperative in 



Figure 2 Direct indicators of assessment in academic departments 
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Figure 3 Indirect indicators of assessment in academic departments 




assessment (Communications) vs. the probable resisters (Political Science). The contrast 
of Arts and Letters with the profiled College of Education (Figure 5) is revealing. 
Education has separate discipline accreditations and is accustomed to demonstrating 
student outcomes. Their profile identifies them as potential cheerleaders or advocates 
whose experiences might be used to energize their colleagues in other colleges. 

A parallel ordering of assessment indicators for administrative and academic 
support units showed a number of indicators already in use (Figure 6). The ones 
generating most interest for fixture use were measures that could be implemented using a 
centrally administered client satisfaction survey, and comparisons with peer institutions. 

When asked what resources were needed to develop better methods for assessing 
outcomes, department chairs consistently mentioned financial and human support, 
training and technology. Assistance with development of instruments and tracking of 
program graduates was frequently requested. Department chairs also expressed concern 
over the time required to identify, gather and produce documentation and information 
supporting departmental efforts for improving student performance-related outcomes. 
They urged establishment of a timeline delineating tasks, responsible individuals, 
resources and due dates to ensure that the process stayed on track. They believed that 
this type of coordinated effort would best be achieved with institution-wide leadership 
and support fi-om the Office of Institutional Effectiveness and Analysis. 

The inventory checklist analysis led us to the sobering realization that we had far 
to go before accreditation time. We promptly arranged for outside consultants to conduct 
a two-day workshop to further acquaint administrators, faculty and staff with the 
concepts of institutional effectiveness and provide expert guidance for those willing to 
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Profile of indicators - College of Arts and Letters 
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get started with their plans. A decision was made to form an Academic Programs 
Assessment Committee with representation from all colleges to oversee the development 
of program assessment plans at all degree levels. Since research has shown that the 
biggest obstacle to implementing student outcomes assessment is faculty resistance 
(Maki, 1999), there was a strong effort to extend Committee membership to senior 
faculty with few or no administrative responsibilities. The Committee will recommend 
approval or modification of the plans to the Provost. The expressed need for technical 
support to assist departments in instrument identification and data analysis resulted in the 
funding of a new position for an assessment coordinator. Meanwhile, the SACS Self- 
Study Committee charged with Institutional Effectiveness reviewed the inventories and 
prepared an initial report for the administration on the compliance status with Section III 
of the Criteria. As the process of developing assessment plans proceeded, the checklists 
served as a baseline for measuring progress. 

Conclusion 

The assessment of student performance-related outcomes offers an excellent 
opportunity for initiating a systematic and continuing process for gathering, interpreting, 
and using information that will result in increasing institutional effectiveness and 
educational improvement. Although a systematic process for assessing student 
performance-related outcomes is mandated by accrediting agencies, the initiative can be a 
catalyst for many positive changes in improving educational outcomes and institutional 
effectiveness. 

For us, the inventory checklist was a way to introduce the concept of student 
outcomes assessment in a nonthreatening, low-risk, collegial context. It allowed us to 



answer questions, allay fears, and confront resistance on an individual basis. Talking in 
person is particularly useful for those who are new to the process. By reassuring faculty 
and chairs that many of their existing efforts were already moving in the right direction, 
we bought goodwill for the project. Reluctant faculty were often pleased to discover that 
assessment did not necessarily involve only quantitative measurements and did not 
require standardized testing. Finally, the inventory results got the attention of campus 
administrators, whose leadership was critical for ensuring that assessment efforts stayed 
on track. 

Two minor limitations of the checklist surfaced during its use. There was some 
initial confusion over the role this information would play in the Self-Study process. 

Some chairs were concerned that the information submitted via the checklist would 
represent their department’s final, rather than initial, status with respect to institutional 
effectiveness and that they would not have an opportunity to update their progress. 
Second, the examples of direct and indirect measures on the checklist, while 
comprehensive, were not intended to preclude other kinds of measures that might be 
devised for program assessment. There was some risk that faculty and administrators 
would not look beyond these indicators for others that might be more appropriate for their 
programs. 

Successful assessment is more than a collection of techniques, instruments and 
outcomes; it is a cultural issue that affects how a community of scholars defines its 
responsibilities to its students (Magruder, McManis & Yoimg, 1997). Now that we’ve 
identified the ‘Start Line’, the goal of transforming the information collected via the 
inventories into a successful assessment culture seems more attainable. 
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INSTITUTIONAL EFFECTIVENESS CHECKLIST 

For Academic Departments 




College:: 
Department: 
Program (Major): 
Level: 




Part I. Does the program listed above have: 

1 . A written mission or statement of purpose? 



4 . 



O 

ERIC 



Yes 



No 



If yes, please attach a copy or reference a website and/or catalog for retrieval of this information. 



Statements of intended educational outcomes? 

(This term describes what the departmental faculty intend for a student to be able to think, know, or do when they’ve 
completed a given educational program.) 



Yes 



No 



Written methods of assessment for evaluating program effectiveness in terms of measurable student outcomes? 

Yes No 



A separate accreditation agency or process? - 
If yes, please list all accreditation agencies. 



Yes 



No 






Part II. Assessment of Outcomes: 



During the past year, has your program used any of the following for assessment of outcomes? 

Indicate “A” if currently being used; “B” if not currently being used but interested in using; and “C” if not applicable. 

Direct indicators of assessment: 



1. 

2 . 

3 . 

4 . 

5 . 

6 . 

7 . 

8 . 
9 . 

10 . 

11 . 



12 . 




Comprehensive exams 
Writing proficiency exams 

National exams assessing subject matter knowledge (e.g., Major Field Achievement Test) 

Graduate Record Examination (GRE) subject test 
Certification exams 
Licensure exams 

Locally developed pre-test or post-test for mastery of knowledge 

Performance assessment for graduating seniors (i.e., recitals, art exhibits, science projects, etc.) 

Video and audio tape evaluations (i.e., music, art, student teaching, etc.) 

Senior thesis/major project 

Portfolio evaluation containing representative examples of student’s work 
(i.e., written, creative, or scientific papers or projects) 

Capstone courses which are designed to measure student mastery of essential theoretical and 
methodological issues associated with a discipline (e.g., senior level seminars) 

/ 



Indirect indicators of assessment: 



1. 

2. 

3 . 

4 . 

5 . 

6 . 

7 . 

8 . 
9 . 

10. 

11. 

12. 

13 . 

14 . 

15 . 

16 . 

17 . 

18 . 

19 . 

20 . 
21 . 



Comparison of outcomes with peer institutions 

Job placement of graduating students 

Employer surveys and questionnaires 

Graduate school acceptance rates 

Performance in graduate school 

Student graduation/retention rates 

Exit interviews 

Student satisfaction surveys 

Student course evaluations 

Internship evaluation 

Focus group discussions 

Alumni surveys reporting satisfaction with degree program and career success 
Tracking of alumni honors, awards, and achievements at local, state, and national levels 
Identification and assessment of at-risk students 
Analysis of student grade distributions 

Examination of information contained in department’s own database 
Other evaluations of course instruction (e.g., chair or peer review) 

Curriculum/syllabus analysis (e.g., analysis of transfer student preparation) 

Community perception of program effectiveness 
Community service/volunteerism participation 
Other: 
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Part III. Other Information 



I . Has your department used any of the indicators listed above to improve departmental programs, services, and 
operations? Yes No 

If yes, please identify some examples. 



2. What resources (i.e., training, personnel, technology, etc.) does your department need to develop better methods 
for assessing student outcomes and improving program effectiveness? 



3. Please list any additional comments or concerns. 




Completed by: 



Date: 



INSTITUTIONAL EFFECTIVENESS CHECKLIST 
For Administrative and Academic Support Units 

Unit; Campus: 



Part I. Does your unit have 



1 . A formal statement of purpose which supports FAU’s mission and goals? 

Yes (please attach a copy) No 

2. Explicit goals which support this unit’s purpose? 

Yes No 



3. Procedures to evaluate the extent to which goals are being achieved? 




Part II. Evaluation Measures 



During the past year, has your unit used any of the following for assessment of outcomes? 
Indicate “A” if currently being used; “B” if not currently being used but interested in using; and 
“C” if not applicable. 

1 Measures of volume of activity 

Examples: Number of clients served, circulation data, gross sales. 



Specify: 



2 Measures of efficiency 

Examples: Average turnaround time for filling requests, timely service/prompt response, budget information. 
Specify; 



3 



4 



5 




Measures of service quality 

Examples: Error rates, accuracy of the information provided. 

Specify: 

Client satisfaction surveys 

Examples: Student satisfaction survey, alumni survey, employer survey, customer survey. 
Specify: 



Other methods to obtain client feedback 

Examples: Focus groups, comments via email, evaluation fonns, suggestion box, hotline. 



Specify: 






Part II. Evaluation Measures - continued 



6 . 



7.. 



, Staff discussions/evaluations of services to clients 
, Review of existing data 

Examples: Departmental routine records/reports, institutional data, audits. 
Specify; 



8. Standards/guidelines provided by professional associations such as SCUP, NACUBO 

9. Standards set by federal, state, county, city or FAU regulations 

10 External evaluators/auditors 

11 Benchmarks/Comparisons with peer institutions 

12 Other: 

Part III. Other information 



1. Have you used the results of any of the evaluation measures listed above to improve administrative and academic 
support services and operations? Yes No 

If so, please identify some examples. 



2. What resources (i.e., training, personnel, technology, etc.) does your unit need to develop better methods for assess 
ing service outcomes and improving service quality and effectiveness? 



3. Please list any additional comments or concerns. 



Completed by: 



Date: 
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Please return your completed form to: 

Institutional Effectiveness and Analysis 
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